From Generation to Suppression: Towards Effective Irregular Glow Removal for Nighttime Visibility Enhancement

1 Introduction

Images captured in low light are often accompanied by reduced visibility and information loss in dark areas. Thus, low light image enhancement (LLIE) has sparked a surge of interest. Apart from dim light's impact on image quality, the widespread presence of glow effects around light sources significantly reduces visibility in real-world nighttime scenes. However, existing LLIE methods devote to low-light areas brightening only and are not well-suited to tackle glow effects. When applied to images with glow, these methods may actually exacerbate the problem, causing further diffusion of glow areas and introducing artifacts that degrade overall image visibility (Fig. 1). To this end, developing effective glow suppression techniques is an urgent task.

∗Corresponding author: wangwei8@wust.edu.cn

Our REM

Figure 1: Nighttime images suffering from glow effects will be further diffused after existing LLIE methods, significantly impairing the visibility. The proposed LBDN and REM can effectively handle glow effects and boost nighttime visibility.

Deep learning has proven promising results for LLIE tasks [Wang et al., 2020; Jiang et al., 2021; Li et al., 2022], while a substantial portion of which incorporates the traditional Retinex model [Jobson et al., 1997] for better luminance boosting through suppression of noise. To make a step forward, a large body of work has progressively exerted great effort in more extreme conditions [Chen et al., 2018; Zhang et al., 2019a; Zhu et al., 2020b; Wei et al., 2020]. Regretfully, all these methods cannot produce a satisfactory performance on nighttime images with glows.

In a recent study [Sharma and Tan, 2021], the first enhancement method for suppressing glow effects is proposed. This method regards the glow as low-frequency information in the image and adopts a decomposition model [Wu et al., 2018] to disentangle the linearized image into low and high-frequency feature maps. In a similar way, the glow in [Jin et al., 2022] is removed via high-low frequency decomposition on the assumption of smooth glow in the low-frequency layer. However, these methods do not consider the fact that glow regions may occur in both low-frequency and high-frequency layers, as is often observed in captured images.

Glows demand a more universal description to form a reliable solution. To form a universal definition of the glow suppression task, this study defines the glow based on its physical formation, which arises from a light source scattered several times across medium particles characterized by the atmospheric point spread function (APSF). The first attempt to achieve glow suppression from glow generation is introduced via deep learning-based APSF formation.

Real-world glow effects pose two challenges, the complex variety of light source shapes and the glow intensities unevenly decaying from light sources, as depicted in Fig. 2, featuring rough or divergent glows. To solve these challenges, our nighttime imaging model with near-field light sources (NIM-NLS) is specifically tailored to account for all multiscattering lights associated with arbitrary glow effects, direct reflectance from objects, and the impact of ambient light.

Leveraging the physical model, we proceed to solve the problem of recovering both the blurred glow and the clear transmission in NIM-NLS. However, we encountered further challenges, which highlighted limitations in existing methods. Specifically, we observed that: 1) glow only affects local regions around the light source, existing global restoration models cannot recover glow region accurately [Zhou and Susstrunk, 2019; Ren et al., 2020; Feng et al., 2021]; 2) glow patterns are irregular and not sharply bounded, which cannot be extracted by layer separation methods [Levin and Weiss, 2007; Zhang et al., 2018; Gandelsman et al., 2019]. Drawing inspiration from these findings, our light-aware blind deconvolution net (LBDN) is designed to construct a local glow estimation. Guided by the principle that the center of the glow exhibits maximum intensity, our approach involves initially segmenting the light sources, followed by performing a blind deconvolution on the local glow. Irregular glow formulation is introduced in our approach with two priors. The light source spatial location mask prior M0 indicates spatial locations, while the learning-based APSF prior accommodates for varying glows. Notably, our LBDN can be flexibly extended into existing LLIE methods as a pre-processing module.

To the best of our knowledge, the proposed method is the first to tackle visibility degradation in various forms of uneven and irregularly shaped glows (see Fig. 3). In the meantime, our approach is not reliant on paired or unpaired images, nor does it necessitate pre-training. In summary, our contri- Input I Multi-scattering G Transmission D

Figure 3: Examples obtained from our LBDN, demonstrating the effective removal of irregular glow. G is the multi-scattering map estimated by a network, while D is the clear direct transmission map after removing the multi-scattering.

butions can be concluded as:

• Perspective contribution. The glow suppression task is innovatively treated as the learning of glow generation, where learning-based APSF formation is utilized.
• Technical contribution. Two challenges of arbitrarily shaped light sources and glows with uneven intensity are tackled for the first time. A novel nighttime imaging model with near-field light sources is derived, and solved by a scalable light-aware blind deconvolutional network.
• Practical contribution. Our effectiveness on glow suppression and scalability in existing LLIE methods are validated on real-world datasets. Meanwhile, the proposed method does not require any pre-training.

2 RelatedWork

In this section, we outline relevant works on deep learningbased methods for LLIE and light effects suppression approaches in fields such as nighttime dehazing.

2.1 Low-light Image Enhancement

The LLIE task aims to increase image visibility so as to benefit a series of downstream tasks (e.g. classification, detection, and recognition). Benefiting from the flourishing of deep learning, a large proportion of existing LLIE methods provide outstanding results in typical low-light scenes. Part of these methods [Zhang et al., 2019b; Zhang et al., 2020b; Zhang et al., 2021] typically introduced the Retinex model [Jobson et al., 1997] from traditional methods, enhancing the luminance and reflection components separately through dedicated subnetworks. On another matter, the elimination of noise, an explicit term in the imaging system, is often decoupled into a separate sub-task [Zhu et al., 2020a; Yang et al., 2021; Hai et al., 2023; Guo and Hu, 2023].

Furthermore, more recent attention has focused on realistic nighttime scenarios. On the one hand, real night scenes lack paired datasets, hence EG [Jiang et al., 2021] and DCE [Li et al., 2022] escaped the reliance on paired datasets by utilizing an unsupervised generative adversarial network and designing image-specific curves, respectively. On the other hand, some approaches [Zhang et al., 2019a; Zhu et al., 2020b] attempted to confront more challenging light conditions like extreme low-light [Chen et al., 2018; Wei et al., 2020].

However, all these methods fail to address the extensive and blurred glow caused by nighttime light sources, which can be further exposed with direct enhancement, thus resulting in even more information loss.

2.2 Glow Suppression

The primary object of glow effects suppression is the glow from multi-scattered near-field light sources (distinct from atmospheric light). Due to the scarcity of glow suppression methods in LLIE, we conduct research in adjacent areas.

Generally, the glow formation function is referred to as APSF. Since glow causes a specific range of information loss, modeling APSF is essential to improve the performance of outdoor vision systems. [Narasimhan and Nayar, 2003] introduced APSF into computer vision for the first time and designed a physics-based model to describe multiple scattering. As it is well suited for severe weather such as fog, haze and rain, [Li et al., 2015] brought the model into nighttime haze removal and utilized the short-tailed distribution of glow on the gradient histogram to separate the glow layer, followed by [Park et al., 2016; Yang et al., 2018]. Lately, in response to noise, flare, haze, and blur in Under-Display Camera (UDC) images, [Feng et al., 2021] defined a physicsbased image formation model to analyze and solve the aforementioned degradations. Besides, the authors measured the real point spread function (PSF) of the UDC system and provided a model-based data synthesis pipeline to generate realistic degraded images. But the specific glow pattern in the UDC system is not applicable in a wide range of real scenes, and it is not feasible to measure true PSFs in each case.

In LLIE, [Sharma and Tan, 2021] is the first to focus on glow effects suppression. More recently, the same research team introduced an unsupervised method [Jin et al., 2022] that integrates a layer decomposition network and a lighteffects suppression network. Yet a rudimentary frequencybased decomposition struggles to effectively separate glows from irregularly shaped light sources or those with uneven intensity. This is due to the absence of interpretable physical model support.

In contrast, the proposed method is fully theoretically informed, as evidenced by a physical model designed on glow formation, thus allowing for the effective suppression of various shapes and intensities of glows shown in Fig. 3. Moreover, unlike the above works requiring training datasets with and without glow, our approach does not rely on any paired or unpaired datasets, nor does it require pre-training.

3 Physical Model

3.1 APSF-based Physical Glow Formation

As the glow formation described in Fig. 4, rays from the light source are scattered several times by particles, and further interactions with different angles overlap on the image to form glow. Finally, the glow in the image plane forms a round region, which has a maximum intensity in the center point and decreases towards the surroundings.

To obtain the multi-scattering map of irregular light sources, the imaging of glow is derived by the intensity of the multiply scattered radiations I(T, µ) from a point source with APSF [Chandrasekhar, 2013], which is related to the forward scattering parameter q, the phase function P(cosα) of the particles in the atmosphere, and the optical thickness of the atmosphere T. The solution to I(T, µ) is given by expanding the Henyey-Greenstein phase function [Ishimaru, 1978] in terms of Legendre polynomials. Please see the supplementary material for more detailed formulations.

The glow around one light source of arbitrary shape S is formed as a combination of multiple isotropic light source elements with different radiation intensities I0(x, y). With light from each source element approximately passing through the same atmosphere and, based on the superimposability of light, the intensity I0 of different light source elements can be summed to yield the overall intensity Itotal :

Accordingly, the connection between the image P with ideal light sources and the final imaging result G after suffering from multiple scattering can be established as:

In this paper, we take advantage of deep learning to estimate the APSF via a kernel estimation network to enable the injection of this physical prior into our network as guidance.

3.2 Nighttime ImagingWith Light Sources

In this section, we refer to existing physical models from neighboring domains to design our nighttime imaging with near-field light sources. In bad weather, the light source contributes to two light components, one is the light reflected from the target through particle attenuation, and the other is the atmospheric light formed by the scattering of ambient light in the infinite distance, in correspondence to the direct transmission D(x, λ) and atmospheric light A(x, λ) in the atmospheric scattering model, with x and λ representing the position of the pixel in the image and the wavelength of light:

where I(x, λ) is the degraded image captured by a camera and R(x, λ) denotes the ideal image to restore; L∞ represents the value of atmospheric light at infinity; and t = e-ˆ β(λ)d(x) represents the transfer function with the physical meaning referring to the proportion of light, which reaches the image plane via attenuation light through atmosphere particles.

While in the inference of our model upon Eq. 3, the following three items were sequentially reconsidered:

1. Atmospheric light: Contrary to the atmospheric scattering model aiming at imaging bad weather, we serve for the LLIE task, whose target scene is a clear night. As the nighttime atmospheric light value L∞ and the total scattering coefficient ˆ β(λ) in a clear night both converge to zero, the atmospheric light A(x, λ) in our model can be safely ignored.
2. Multi-scattering: Since D(x, λ) is the direct transmission after removing the scattered flux of all incident energy, excluding the glow formed by scattering, which is of our concern. For that, we additionally consider the prevalence of active light sources at night and represent the result of their multiple scattering as G(x, λ) addition to the model, with its modeling equivalent to Eq. 2.
3. Direct transmission: Inspired by Retinex theory [Jobson et al., 1997], our transmission map D reaching the observer can be decomposed into an illumination map L and an illumination-independent reflectance R. Such decomposition helps to subsequently correct the illumination conditions individually for better visibility.

Adapting Eq. 3 to the above derivation, our final nighttime imaging model NIM-NLS can be expressed as:

where P is the light source map indicating the positional information perceived from light sources in glow.

Our physical imaging model, NIM-NLS, describes the actual generation of nighttime glows without any assumptions or restrictions on the intensity, and in which APSF is proven capable of modeling the scattering of arbitrarily shaped light sources, effectively addressing the two challenges of realworld glows (Fig. 6).

4 Network

To solve for NIM-NLS, we propose a network (Fig. 5) incorporating a Light-aware Blind Deconvolution Net (LBDN) and a Retinex-based Enhancement Module (REM), with the former aiming to obtain the glow-free D, and the latter to restore the low-light D for an enhanced one.

4.1 Light-aware Blind Deconvolution Net

On the basis of the Maximum a Posterior (MAP), our objective can be reformulated for deep learning from Eq. 4 as:

where ∥P⊗APSF+D-I∥2 is the fidelity term, and ϕ(D), φ(P), ρ(APSF) are three regularization terms, corresponding to constraint the generation D, P , and the APSF, with the relevant functions specified in Loss Function. λ, τ , and ω are regularization parameters.

The task of estimating the blur kernel and solving for the sharp image is commonly known as blind deconvolution [Shan et al., 2008; Almeida and Almeida, 2009]. Unlike existing global blind deconvolution models, our model should first decompose the light source P locally present in I and then simulate the glow with the APSF obtained from the kernel estimation network. Hence, we design a Light-aware Blind Deconvolution Net (LBDN) to reach achieve Eq. 5, motivated by Double-DIP [Gandelsman et al., 2019] decomposing of a complex image into two or more simple layers. In this case, two DIP generators [Ulyanov et al., 2018] are applied for the light source map P and the direct transmission map D from random noise. Besides, we additionally introduce two priors. One is a mask M0 that marks the shape and location of the light source. The mask derived by setting an illumination threshold of a certain intensity constrains the generation of D and P . The other prior is the APSF, acquired via iterative convolution n times with the blur kernel k from the blur kernel estimation net. It guides the simulation of the multi-scattering map G. Since DIP is suitable for capturing statistical information from natural images while limited in estimating blur kernel priors [Ren et al., 2020], we generate the kernel via a simpler full-connected net (FCN).

4.2 Retinex-based Enhancement Module

The Retinex model has been proven valid throughout extensive previous LLIE works [Zhu et al., 2020a; Zhang et al., 2020b; Liu et al., 2021; Wu et al., 2022]. In the Retinexbased Enhancement Module (REM), we decompose the direct transmission map D reaching the observer into reflection R and illumination L according to Retinex theory. R is always constant under different illumination conditions, so we correct the illumination by gamma transformation for better visibility. In particular, our REM and LBDN are co-related in that they maintain a consistent zero-shot learning approach without any pre-training, which is achieved through a series of non-reference loss functions.

4.3 Loss Function

Our total loss function consists of the generation constraint and the enhancement constraint. The former guides the generation of individual layers in the decomposition, and the latter ensures the illumination recovery of the final output.

Generation constraints. Decomposition loss LDec measures the absolute difference between the recomposition of the different layers obtained from Eq. 4 and the input image by constraining the sum of the decomposition layers to fit all the information in the original image as closely as possible.

In addition, we use the generation loss LGen [Gandelsman et al., 2019] to guide the construction of P and D, with the light source spatial location mask M0 serving as prior information for the location of light sources.

Enhancement constrains. In REM, the Retinex loss LR shows the absolute difference between the recomposed D′ from REM and the output D from LBDN. Since the maximum channel of the reflectance coincides with that of the original image, we add this constraint to separate the reflectance.

To ensure that the texture of R is clear, we also introduce the texture enhancement loss LTex in [Zhu et al., 2020a] to strengthen the smoothness of L.

We multiply each non-reference loss with its respective weight, where the weights of generation constraints are 1.0, while enhancement losses' are set to 0.5.

5 Experiment

5.1 Experimental Settings

Implementation details. Our implementation is done with PyTorch on an NVIDIA GeForce RTX 3060 GPU. Since our model is based on zero-shot learning, which trains and tests each individual input to obtain the specific glow-free result, pre-training is not required.

Datasets. We test on two groups of real-world datasets. The light-effects dataset [Sharma and Tan, 2021] contains 500 images with multiple light colors in various scenes. Besides, we take 30 images with glow in different shapes and uniform intensity with Huawei Note 20 and Redmi Note 10 Pro.

5.2 Comparison on Glow Suppression

We show the effective suppression of a wide variety of glows in Fig. 6. Our method performs excellent suppression when confronted with glows from different scenes, such as streetlights and headlights with a very wide luminous area, fireworks with dense and unevenly distributed light points, and billboards with irregular shapes.

To further show the superiority of the proposed method in improving nighttime visibility, we compared it with two existing glow suppression methods, UNIE [Jin et al., 2022] and Sharma[Sharma and Tan, 2021]. As there is minimal work on glow suppression, supplementary consider three nighttime dehazing methods, including 3R [Zhang et al., 2020a], NHR [Li et al., 2015] and DCP [He et al., 2010]. The subjective results are in Fig. 7. Our method provides effective suppression in regions of uneven intensity or high frequencies in the glow, as evidenced by the zoomed-in local glow zone in Fig. 8 together with the heat maps (more visual results can be seen in the supplementary material). In Table. 1, two metrics from dehazing are borrowed to test the clarity of glow patches in each image. The metrics e and r [Hautiere et al., 2008] indicate, respectively, the ratio of new visible edges and the percentage of saturated pixels produced. For the reconstruction quality of the whole image, we believe a user study is highly recommended, given the lack of recognized objective metrics for glow suppression. Therefore, 20 participants were involved in ranking these methods based on 1) glow suppression effectiveness, 2) visibility, and 3) the presence of color bias and artifacts, with 1 to 7 in order of worst and best.

5.3 Low-light Image Enhancement

Our initial aim in suppressing glow effects is to address the problem of further spread of glow in LLIE resulting in sharpness degradation or even color cast and artifacts. To validate our scalability, we combine the existing SOTA methods in LLIE to compare the performance before and after using our LBDN. The chosen methods includes KinD [Zhang et al., 2019b], Zero-DCE [Li et al., 2022], EnlightenGan [Jiang et al., 2021] and Bread [Guo and Hu, 2023]. The subjective and objective results on the light-effects dataset [Sharma and Tan, 2021] are presented in Fig. 9 and Table. 2 respectively, indicating that our LBDN significantly moderates the loss of information caused by further diffusion of glow during enhancement, as can be seen from the license plate in the first set of images and the ground in the second set. More display results can be found in the supplementary material.

5.4 Ablation Study

In this section, we demonstrate the effectiveness of some modules and parameters of the proposed method through ablation experiments. The subjective results and corresponding objective indicators are shown in Fig. 10 and Table 3.

Iterative convolutions for APSF. To fit the APSF, we approximated the optimal solution by iterative convolutions, considering that the size of the convolution kernel limits the range of the simulated glow, whereby the number of iterations is represented by the parameter n. Although the objective index is not optimal when n is taken as n = 3, we believe that the glow suppression and overall visibility improvement are better when combined with the visual outcome.

APSF prior generator (APG). Our APSF prior generator serves as the core for glow suppression with direct decisions on the generation of D and G. When removing the APM, we degrade the model to the basic Double-dip, which targets the light sources instead of irregular glow and offers almost no effect on glow suppression.

Light mask prior (LMP). Since glows are present locally in the image, we perform light source segmentation by a light mask prior (LMP) before blind deconvolution. As shown in Fig. 10, without LMP, the light source cannot be located accurately and thus the removal of the glow is very limited.

Loss function. The key parameter M0 (LMP) in generation losses has already been discussed, and the importance of composition losses is apparent since their removal will lead to the lack of principle for decomposition. Hence, we only focus on the ablation of LTex and Fc(I, x) here. Removing LTex makes the decomposed illumination map patchy, resulting in unnatural artifacts in the output. While the absence of Fc(I, x) invalidates the decomposition of the illumination map for proper light correction.

6 Conclusion

To remove irregular glows for nighttime visibility enhancement, we propose a novel approach for the first time viewing the glow suppression task as the learning of glow generation. We formulate a nighttime imaging model with near-field light sources, NIM-NLS, in which the APSF was first introduced into deep learning to learn the glow term. To fit this physical model, we accordingly design a light-aware blind deconvolution model LBDN and a subsequent Retinex-based enhancement network REM. The multi-scattering map can be estimated with the light source spatial location mask and the learned APSF as priors. Later in REM, the illumination of the clear transmission map is adjusted for better visibility. The proposed method does not rely on any paired or unpaired data, even without pre-training, and we demonstrated the method's effectiveness in extensive experiments.

Acknowledgments

This work was supported by the Natural Science Foundation of China (62202347) and the Natural Science Foundation of Hubei Province (2022CFB578).